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ABSTRACT 

Guidelines for evaluation of school programs in the 
State of Florida are presented. The purpose of evaluation is to 
modify the content and method of instruction, provide information to 
decision-makers and the public, provide feedback to participants, and 
determine criteria for program success. The optimal evaluation design 
for either innovative or reference programs is the pre-test/post-test 
design. Instruments that may be used in evaluation include tests 
^norm-referenced, criterion-referenced, or objective-referenced), 
rating scales, questionnaires, and observation forms. The types of 
data generally collected include sunmiative product data, formative or 
process data, and context data. Other evaluation issues are 
associated with data-collection procedures, data analysis, monitoring 
quality, reporting findings, and use of findings. Four instruments 
for use as resource tools in planning and implementing the evaluation 
process are appended; (1) a program evaluation planning form; (2) a 
checklist of the major steps in developing an evaluation plan; (3) 
selecting norm-referenced tests; and (4) the Evaluation Checklist. 
(TJH) 
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Please direct all questions and comments to the Division of Public Schools, 
Bureau of Curriculum S^vices, Early Childhood and Elementary Education 
Section, at (904) 488-4888, Suncom 278-4888. 



FOREWORD 



Evaluation of school programs has become a necessity for all school districts. 
With greater accountability of financial resources, decisions on how to most 
effectively cdlocate available funds must be made. While basic educational 
research provides much valuable information, that information is usually not the 
kind on which day-to-day decisions about specific educational programs are 
based. Program evaluation is a means by which useful information is collected 
and analyzed by a local educational agency for decision making. The purpose of 
this Program Evaluation Guide and Ev aluation Checklist is to assist school 
district personnel such as primary specialists, curriculum supervisors and 
principals in planning for and conducting evaluation of districts and/or school 
programs. 




Evaluation is an integral part of the program planning process. It is at this stage 
that the program purpose, goals, objectives, evaluation design, assessment 
instruments, data, and use of evaluation results are determined. The function of 
program evaluation is the improvement of instruction. Curriculum personnel 
and evaluation personnel should work together and with teachers to assure 
quality evaluation. 

The appendix of this document includes four instruments which can be used as 
resource tools in the planning and implementation of the evaluation process. 

APPENDIX A - PROGRAM EVALUATION FORM is designed to focus on key 
points which need to be considered in planning for program evaluation. 
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APPENDIX B - CHECKLIST OF MAJOR STEPS IN DEVELOPING AN 
EVALUATION PLAN can be used to record the mtiestones in the development of 
an evaluation plan. 

APPENDIX C - SELECTING NORM-REFERENCED TESTS organizes needed 
information in selecting norm*ref erenced tests. 

APPENDIX D - EVALUATION CHECKLIST is a step-by-step guide to evaluation. 
The explanations and descriptions of each task are consistent with those in this 
Program Evaluation Guide. Each task should be initiated as indicated in the 
checklist. 

Both the Program Evaluation Guide and the Evaluation Checklist are based upon 
materials developed by Educational Testing Services, Princeton, New Jersey. 
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Program evaluation is a continuing process. One may consider evaluation as the 
nucleus of the program since it interacts with the program^ needs assessment, 
statement of goals and objectives, and planning and implementation. The entire 
activity is cyclic, as iUustrated below: 

Evaluation is an Integral Part of a Program 



Why^ 



Needs 
Assessment 



What needs can be cited that 
justify the existence of 
this program? 



What? 



Howl 



How Will 
You Know? 



Program Goals 
and Objectives? 



Program 
Activities 



Program 
Evaluation 



What are the goals to which the 
objectives pre related? 

What are the goals addressed by 
the objectives? 



What activities wiU most 
likely meet the objectives? 

How will planning and 
activities be carried out to 
accomplish the objectives? 



What kinds of information 
should be gathered to 
determine if the activities 
are reaching the objectives 
and consequently meeting 
the needs? 
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**Evaiuati^** is a term used to describe a variety of activities from test 
administration to conducting complex research projects. In this document, 
evaluation is described as the process of determining the value or effectiveness 
of an activity for the purpose of decision-matdnr* Decision-making refers to 
determinations regarding continuation, modificaticni and discontinuation of a 
program. 

This Program Evaluation Guide presents an overview of evaluation tasks listed in 
the Evaluation Checklist. Of the ten topics which are included in the guide, 
eight are task headings in the Evaluation Checklist anu have been underlined for 
identification in the following list: 



0 


Purpose 


0 


Goals and Objectives 


0 


Evaluation Design 


0 


Assessment Instruments 


0 


Data Requirements 


0 


Data Collection 


0 


Data Anal^^is 


0 


Monitorine 


o 


Reporting 


0 


Use of Evaluation Results 



PURPOSE 

Evaluation is Jucted to determine whether a program is having the effect it 
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is supposed to have. Two general questions provide a focus for the evaluation of 
any educational program; 

o How is the program being implemented? 

o Are the results of the program appropriate? 
The first question requires data associated with the process of program 
implementation. Analyses of data may be used to change the educational 
program so that it is more likely to achieve its goals and objectives. 

The second question requires product data. Answers to this question are likely to 
be used as a basis for decisions about continuation, modification or 
^discontinuation of the program. 

Therefore, the data analyses results may be used to: 

o modify the content and method of instruction; 
o provide information to decision-makers; 
o communicate with the public; and, 
o provide feedback to the participants. 



Modify the Ckmtent and the Method of Imtiuetion 

Effective evaluation provides ongoing information about a program. This 
includes information about the extent to which the content and method are 
consistent with the program plan. It also includes information on the attainment 
of interim goals and objectives. Thus, program evaluation can examine questions 
such as "Is the program being implemented as planned?" and "Should elements of 
the program (either content or method) be modified or excluded?" 
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information to Decision-Makers 

Eval'jation can be useful in providing information to decision-makers at all 
stages of program planning and implementation. It is essential, however, that 
the information communicated be in response to the decision-makers* questions. 
To provide information which does not answer the questions of decision-makers 
reduces the probability that the evaluation findings will be used. 

Cominunieation with the PuUie 

Evaluation can be used to provide information to groups or to individuals. 
Reports to the public should be based on a full range of program objectives and 
should show the extent to which the objectives are realized. When this is 
accomplished, the public will be able to make iiiore informed juc^ments about 
the effectiveness of school programs and what is needed to gain support for 
them. 

Provide Feedback to the Participants 

Evaluation can provide information to the participants. Information about an 
individual's progress in the program can be used for more appropriate instruction, 
individualization and remediation. Evaluation can also provide participants with 
a regular and systCiTiatic method of expressing opinions about the program. 

Determining the Purpose 

A review of the four uses of evaluation will lead to the identification or 
development of a dear statement of purpose for the evaluation. That is, the 
decision-makers^ instructional planners/curriculum developers, other participants 
and the public, (these pers*^ ns who are likely to have an impact on the way the 
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evaluation results are used) are brought to accept the purpose. It is important 
that agreement is reached among all interested parties as to the purpose and its 
reasonableness and that legitimate evaluation practices will address that 
purpose. 

Determining Criteria For Program SuecesB 

When the purpose of the evaluation has been determined, there must be 
agreement on the criteria for program success. These criteria may be general in 
nature, such as: 

o the student will master 80% of the content 
on a given test at post-test time 
The designated users of the evaluation results must agree on the criteria for 
program success before the evaluation begins. Without agreement among the 
users regarding what indicators of program "success" are acceptable, there Vj 
little chance that reasonable and consistent interpretation of the data will 
emanate from different users. Committee review is one useful method for 
obtaining agreement. 

GOALS AND OBJECTIYES 

After the purpose of the evaluation has been established, it is the responsioility 
of those involved in the program and its evaluation to: 

o distinguish between goals and objectives; 

o determine whether the goals and objectives are 
compatible; and, 

o determine whether the objectives and the evaluation 

questions (derived from the purpose) are compatible. 
-7- 
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To obtain explicit goals £Lnd objectives, it may be necessary to clarify for the 
evaluation users that goals and objectives are different. While goals are broad 
general statements, objectives are narrow specific statements. Furthermore, 
goals are not directly measurable. Objectives are directly measurable. 



EXAMPLE OF A GOAL STATEMENT 

Students should acquire the academic skills required for 
study. 

EXAMPLE OF A PROGRAM OBJECTIVE 

Sixty percent of the 4th grade students participating In 
the KLM Reading Program will score at or above the 50th 
percentile on the Coiuprehensive Test of Basic Skills after 
eight months of specialized reading instruction. 



The goals and objectives must be compatible. Once the compatibility between 
goals and objectives has been determined, the objectives must be reviewed in 
terms of the evaluation question s. The objectives may need to be rewritten so 
that they are observable. The rewritten objectives should be reviewed. A well- 
stated objective contains six components that will answer the following 
questions: 

o Who? 

o Learns or does what? 
o When? 

o Under what eonditions? 
o At what performance level? 
o How will it be measured? 
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The Wio relates to the person who performs an activity. The Learns or does 
what is the activity to be performed. When, Under wtmt conditions, and At what 
performance level relate to time and performance conditions. How will it be 
measured relates to assessment techniques. 

EVALUATION DESIGN 

The evaluation design is essentially a systematic approach to the task of 
gathering information to answer questions or make decisims. The design cannot 
be develor <1 until the program gails and objectives have been clarified (e.g., 
increased achievement, more positive effect among pupils and/or staff, retained 
achievement, reduced per pupil expenditure). Furthermore, the constraints 
associated with the evaluation activities (e.g,, limited time, money and 
expertise), must be known. Once the purrH>se of the program and the constraints 
on evaluation activities are clear, the task at hand is to design an evaluation 
which will address appropriate questions and provide both reliable and valid 
information. 

The type of aesign selected for evaluation depends oi^. the evaluation questions 
that are asked. For example, if the evaluation design is limited to the question 
"Is the program accomplishing its objectives?" an objectives-based approach 
(following Tyler, 1971) is used. If, however, the question is "Does the new 
pr^t^.»am produce more or greater improvement or change than does the existing 
program?" an experimental design is more appropriate. With the objectives- 
based approach one of two evaluatim designs can be used. These are a "post- 
test-only" or a "pre-test/post-test" design. 
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In either choice, the evaluation will determine whether the summative objectives 
of the program are met. In the post-test only design, no information f.s available 
^ut the level of achievement of the objectives before the treatment begins. 

A pre-te3t/post-test design requires that information be generated about t^^ 
level of achievement of the objectives both before the treatment and after the 
treatment. If the evaluation question has to do with the comparison of one with 
another or '^How much more did pupils gain by participating in the program than 
they would have learned without it?*^ an experimental design of some type is 
necessary. Generally, two pieces of information are needed to answer these 
questions. First, one must know how many of the pupils improved between 
beginning and end of tlte program. Secondly, an estimate is needed of how the 
pupils would have progressed in that same amcant of time in another program. 
One approach to this type of evaluation design is to identify a reference group to 
provide the "no-treat menf estimate and to take pre- and post-test measures in 
both group "treatment" and "no-treatment." 



PREMEASUREMENT 


INNOVATIVE 


POSTMEASUREMENT 




PROGRAM 




PREMEASUREMENT 


REFERENCE 


POSTMEASUREMENT 




PROGRAM 





Generally, the reference group may be a control group (no treatment), a 
comparison group (alternative treatment), or a norm group (treatment). 
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Regardless of the type of reference group selected, it is important that the 
selected group is similar to the group experiencing the innovative program and 
that premeasurement averages are similar. Similarity with respect to race, sex, 
age, ability and so forth is desirable. The greater the differences between the 
reference group and the treatment group, the less useful is the comparison data. 

ASSESSMENT INSTRUMENTS 

Appropriate evaluation instruments must be selected to provide information 
which will answer the evaluation questions. Instruments which may be used in 
evaluation include tests (norm-referenced, criterion-referenced, or objective- 
referenced), rating scales, questionnaires, and observation forms. 

Each instrument has its own strengths and weaknesses and should be considered 
in the light of criteria developed for a specific evaluation. Some general criteria 
might be: 

o Does the instrument adequately measure what is to be 
measured? 

o Will the instrument yield consistent results at diffp-ent 
times and with different groups? 

o Is the instrument appropriate for the particular population 
in question? 

o Is the instrument easy to administer and score? 

o Is the cost of the instrument, its administration and its 
scoring, reasonable and within the budget? 

o Are the data produced in a form which is interpreted 
easily? 

Finally, logistics may influence the selection of the data collection instruments. 
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For example, the following questions may be asked: 

o Will special inservice training be required to get good 
results? 

o Are personnel available oa the staff or will outside 
personnel be required? 

o Who can do the assessment with the greatest accuracy 
and with the least disruption to the regular school 
schedule? 

With these questions in mind, a list of appropriate instruments for the evaluation 
should be prepared and/or those arens should be identified for which a technical 
review committee should be established to assist in the selection or the 
development of necessary instruments. 



DATA REQUIREMENTS 



In program evaluation, generally three types of data are collected. Summative 
product data focus on the outcomes, results, or products of the program activity. 
The purposes for collecting such information are to measure and assess status 
and accomplishments at the start, during and at the end of the program. 
Sometimes postprogram follow-up is also done. Product data must be related to 
established program goals and objectives. 



Formative or process data focus on the activities and procedures applied to the 
attainment of the desired outcomes. The purpose for collecting such data is to 
provide a means for measurements and assessments which help determine the 
effectiveness of the various program activities. Process data are used to 
monitor an activity or program and to identify and/or predict procedural 
difficulties. As a result the program activities can be modified. 

-12- 
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Context data describes the environment in which the program activities are 
taking place. The context includes facilities, equipment, supplies, rules and 
pclicf Pi, classroom organization, teacher skills and behaviors, attitude and 
support of the principal toward the program, discipline, and scheduling. Context 
data are useful in determining the appropriateness of program objectives. The 
data serve to identify variables which may enhance or impede the attainment of 
stated program objectives. 

The two types of evaluations and three kinds of evaluation data which may be 
gathered can be visualized as follows: 

TYPES OF EVALUATION DATA 



Formative Summative 
(Interim) (End of Cycle) 

PRODUCT DATA 
(Program Outcomes) 

(Learner Changes) 
(Student Growth) 



PROCESS DATA 
(Instructional Variables) 
(Supportive Activities) 



CONTEXT DATA 
(Institutional Data) 
(Learning Environment) 
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Formative and summative evaluation may include product) process, and context 
data, all three of which may be collected during a program cycle or at the enc of 
a given program period. Specific examples for each of these three types of data 
are as follows: 

Product data : The students in the experimental reading program 
have shown a mean gain of ten months for every six 
months of instruction. 

Pr ocess data : The teachers and aides have carried out all the 
enrichment program activities as planned. 

Context data ; The textbooks arrived two months late resulting in a 
delay in the implementation of the mathematics 
program. 

The availability of the resources and the constraints affecting the evaluation 
plan may determine, at least in part, the scope of the program evaluation. It is 
important that evaluation be restricted to the activities which are worth doing 
and which can be done effectively. It may be necessary to develop a compromise 
evaluation strategy. 

DATA COLLECTION 

To ensure that the data collection is conducted effectively, a list of data 
collection procedures appropriate for each instrument selected must be 
developed. The procedures include instructions for the administrator regarding 
the scheduling of the data collection and scoring of data instruments. The data 
collection schedule includes such information as who is to be tested and where 
and when the testing should take place. 

During the initial process of data evaluation, a sample of the scores must be 
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checked since there are many opportunities for error in scoring and data 
transcription. Close attention must be given to checking the norming dates of 
the instrument to ensure accurate comparison with the norming group. 



In the process of developing the objectives and/or arriving at consensus regarding 
the objectives, certain types of data analyses may be specified. However, if 
analyses are not specified, one must select the least complex analysis which is 
both appropriate to the data and to the evaluation objective and is easily 
understood. Often the person responsible for the evaluation will suggest, select, 
compute and interpret the analysis. Great care must be exercised to avoid 
complexity and to do the least sophisticated and yet most comprehensible 
euialyses. This provides the most useful information. 



Monitoring is the quality control activity of program evaluation. Quality control 
has two parts. The first is monitoring the program being evaluated. The second 
is monitoring the evaluation. 

Monitoring of the evaluation is fulfilled by completing an Evaluation Checklist. 
This checklist provides information at a glance regarding evaluation activities, 
schedules, responsible persons and other pertinent information. The Evaluation 
Checklist can be found in Appendix D. 



DATA ANALYSIS 




MONITORING 
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In monitoring the program, the evaluation and program personnel identify 
specific activities to be monitored. This decision reflects activity priorities or 
emphasis. The activities are then organized into a checklist which may be 
modified on the Evaluation Checklist. The monitoring checklist serves to 
standardize program monitoring across all activities. A schedule which considers 
the data collection schedule, activities, important calendar dates, and other 
constraints, must be developed for monitoring. This schedule should be 
circulated among those individuals affected, such as building administrators and 
classroom teachers, in order that scheduling changes can be made to make the 
schedule more convenient for program personnel. Finally, individuals who will be 

responsible for either program or evaluation monitor^g activities need to be 
identified and trained. 



REPORTING 



After the analyses have been completed and certain outcomes have reflected 
statistical significance or educational importance, this information should be 
summarized so that it is easily understood. Methods of presentation are selected 
which are most appropriate for the chosen audience. 



For some audiences, an oral presentation may be the most effective method of 
communicating the results of the evaluation. And still for others, tables, charts, 
graphs, and a brief exposition may be appropriate. 



After the reporting mode has been selected, the writer of the draft evaluation 
report should consider the following questions; 

-16- 
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o What is justified to be said about the results of the 
evaluation? 

o What cautions must be observed? 

o What kinds of remarks must be avoided? 

As a general rule, global statements that the data "prove" the success of a 
program should be avoided. Statistics do not prove anything. Statistics provide 
the basis upon which people make inferences and interpretations. On the other 
hand, educationally significant findings need to be reported wsiether statistically 
significant or not. The personnel responsible for and working with the evaluation 
need to know and report the relative strengths and weaknesses of the various 
instruments used. It is advisable to acknowledge the difference between data 
collection instruments which require people to perform or demonstrate what 
they know as opposed to just asking them to make judgments or offer opinions. 
Judgments, particularly when made about other people, are prone to large 
fluctuations due to differences which exist among people because of their 
varying standards and background influences. 

Once the draft report has been completed, it should be reviewed by appropriate 
personnel. Reviewers can provide invaluable feedback regarding the usefulness 
and clarity of the report. Based on this feedback, the report may be revised. 
And finally, the evaluation report should be disseminated to the appropriate 
audiences. 
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USE OF EVALUATION RESULTS 



To ensure that effective use will be made of the evaluation findings, conclusions, 
and recommendations, it is important to: 

1. Determine all the purposes the program evaluation is to serve. 

2. Make explicit various questions that all users would like to have 
answered to satisfy their program evaluation needs. 

3. Identify the kinds of information that will prove acceptable as 
evidence bearing upon these questions. 

4. Provide interim reports during the progress of the program to 
give early evidence of movement towards program outcomes, 
even if "soft*^ data is being used. 

5. Prepare the final report clearly and succinctly. The data and 
data interpretations should be presented in a manner that will 

« help the reader recall the questions addressed to understand the 
nature and significance of the answers provided. 



O 
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APPENDIX 



INTRODUCTION TO APPENDIX 



Materials shown in Appendices A through D are from sources as designated and 
are provided as a resource for evaluation planners and implementors. 



APPENDIX A - PROGRAM EVALUATION PLANNING FORfVi is a 
program evaluation form designed to focus thinking 
on key points which need to be considered i:. planning 
for program evaluation. (Program Educator^s Guide ^ 
Educational Testing Services, 1979) 

APPENDIX B - CHECKLIST OF MAJOR STEPS is a checklist which 
helps in keeping a record of the progress in 
developing an evaluation plan, (source is the same as 
for Appendix A) 

APPENDIX C - SELECTING TESTS organizes information needed in 
selecting norm*referenced tests, (source is the same 
as for Appendix A) 

APPENDIX D - EVALUATION CHECKLIST is a checklist for program 
evaluation (Educational Testing Services, 1977) 
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APPENDIX A 
PROGRAM EVALUATION PLANNING FORM 



Program 

Purpose(s) of Evaluation 

Audience(s) for Evaluation Reports 



What objective is being evaluated? 

PROGRAM 

OBJECTIVES What is the goal or what are the need statements to which this 
objective relates? 

Is this objective written in such a form that it can be measured? 
Is the implied measure appropriate for the objective? 



What activities are central to the accomplishments of the objectives? 

Mr 7 these activities reasonably be expected to move participants 
toward the objective? 

What evidence is there that these activities are in progress? 
Which activities should be monitored for process evaluation? 



What questions must this design address^ based upon the stated 
objectives and activities? 

What information must this design be able to produce in order to 
answer these questions? 

To what purposes of evaluation do these questions relate? 

What information will the audience accept as evidence related che 
purpose of the evaluation? 



PROGRAM 
ACTIVITIES 



EVALUATION 
QUESTIONS 
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ASSESSMENT kinds of assessment instruments will be most appropriate to 

INSTRUMENTS secure the information required in the design (Norm- or criterion- 
referenced testSi questionnaires, interviews) observations, rating 
scales, long sheets, or narrative reports)? 



ADMINISTRATION During what month or months should assessment take place? 
DATES AND 

PERSONNEL would be the most appropriate person to collect the data? 

Who is responsible for assigning personnel and dates? 



DATA- What kinds of scores will be most useful in providing the information 

ANALYSIS needed, as identified in the purpose and the design? 

TECHNIQUES 

What kinds of data will be most appropriate? 

Who is responsible for assigning personnel and dates? 



MONITORING 

PROGRAM 

ACTIVITIES 



What activities are central to the accomplishment of the objectives? 

What information must be collected to accomplish the purposes of 
the evaluation? 



MONITORING 
DATES AND 
PERSONNEL 



Who will perform the monitoring function? 

How frequently must the activities for this objective be monitored? 
To whom should the monitoring be reported? 



KEY tvho will be interviewed to ensure that reporting dates meet decision 

REPORTING or user requirements? 

DATES 

Who will establish reporting deadlines? 
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What different audiences will receive evaluation reports on this 
objective? 

Have the questions identified by the audiences during the initial 
design 'Jtep been addressed in the evaluation report? 

Have the purposes of the evaluation been accomplished? 



DETERMINING 

HOW THE DATA What activities have been planned to ensure the most effective use of 
REPORTS WILL the evaluatica reports? 
BE USED 
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WHO IS TO 
RECEIVE THE 
REPORT(S) 
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APPENDIX B 
CHBCKUST OF THE MAJOR STEPS 



IN DBVELOPIHQ AN EVALUATION PLAN 



CHECK 



IN DATE 
PROGRESS COMPLETED COMPLETED 



Review needs 
assessment and goals 
and objectives to 
determine their 
interrelatedness. 

Identify the purposes 
for which the 
evaluation is being 
conducted and the 
probable uses of the 
evaluation. 

Review objectives to 
ensure that they are 
written in measurable 
terms. 

Identify the questions 
that must be answered 
at the end of the year 
as indicated by the 
objectives, the 
purposes, and the 
probable uses of the 
evaluation. 

List appropriable kinds 
of instruments to 
gather the information 
required to answer the 
questions formulated 
above. 
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CHECK 



IN DATE 
PROGRESS COMPLETED COMPLETED 



Determine 
approximate dates 
when the various kinds 
of informati. J would 
most appropriately be 
gathered. 

Determine types of 
data-analysis 
procedures that would 
give the most 
appropriate 
information to answer 
the questions 
formulated earlier. 

List which activities 
need to be monitored 
and when. 

List the kinds of 
reports that will be 
made, formative and 
summative; who will 
receive these reports; 
and the dates the 
reports are due. 

For each report, list 
the potential uses to 
be made of the 
information ar J be 
sure that they match 
the information to be 
gathered. 



APPENDIX C 
SELECTING NORM-REFERENCED TESTS 
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INSTRUMENT 1 INSTRUMENT 2 INSTRUMENT 3 
Criteria Name Name Name 



1. Is the instrument a 
valid measure? 



2. Is the instrument a 
reliable measure? 



3. Is the instrument 

appropriate to use on 
the population to 
be assessed? 



4. Does the instrument 
yield objective 
data? 



5. Is the instrument 
easy to administer 
and score? 



6. Are minimum time 
and resources 
required to 
administer and 
score the instrument 



7. Is the administration 
of the instrument 
nondisruptive to 
classroom learning 
activities. 
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Criteria 



INSTRUMENT 1 INSTRUMENT 2 INSTRUMENT 3 
Name ^Name Name 



Will the instrument 
provide data which 
are useful for decision 
making at both the 
classroom level and the 
program-administrative 
level? 



Is the cost of the instru- 
ment reasonable and 
within budgetary 
cmstraints? 



-27- 



30 



APPENDIX D 
EVALUATION CHBCKUST 



The checklist is a ste(^-by-step guide to evaluation. Each task should be initiated 
in the order given. Task completions will not always be in the same order as 
some tasks take longer than others to complete. Each task marked by a dot ( o ) 
should be completed. Each task formed by an asterisk ( ♦ ) should be completed 
only if necessary. 




PURPOSE 



o Obtain a clear statement of the purpose of the evaluation, 
o Obtain consensus on the purpose of the evaluation, 
o Obtain agreement on the criteria for program success. 

GOALS AND OBJECTIVES 

o Obtain explicit goals and objectives. 

o Determine whether the goals and objectives are compatible. 

o Determine whether the objectives and evaluation questions are 
compatible. 

* Rewrite the objectives in measurable terms. 

* Circulate these rewritten objectives to the program director for 
review. 

o Separate formative and summative objectives. 

o Review program activities to determine whether they are consistent 
with program objectives. 



ASSESSMENT INSTRUMENTS 

o List appropriate kinds of instruments to gather information neediid to 
answer the evaluation questions. 

o Select or develop instruments. 



• 



Review instruments for appropriateness. 



o Arrange purchase, delivery and distribution oi instruments. 

o Check quantities, levels and forms of instruments against actual 
needs. 

* Arrange instrument security for length of the evaluation activities. 
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Responsible Date Date Date Completed 

Person(s) Needed Started Completed on Schedule Comments 
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DATA REQUIREMENTS 

o Identify the type(s) of process, product and context data needed to 
answer the evaluation questions. 

o Prioritize the types of data. 

o Determine the resources and constraints which will affect the 
conduct of the evaluation. 

o Advise evaluation users of those resources which are available and 
those that are required. 

* Submit recommendations to evaluation users for reconciling 
discrepancies between resources available and those required. 

DATA COLLECTION 

o Develop a list of data collection procedures for each assessment 
instrument including directions for administration and scoring. 

o Prepare a schedule for data collection. 

o Review instruments and accompanying manuals and other materials. 

o Select and train persons to administer data collection instruments, 
and provide them with their own sets of materials in advance. 

o Arrange for distribution and collection of instruments. 

0 Plan scoring and other data transcription activities. 

o Check answer sheets for completeness and organization. 

o Double-check a random sample of test scores/data reports for 
accuracy. 

DATA ANALYSIS 

o Identify whether data analyses have been specified for each 
objective. 

* Determine whether data to be collected and analyses are appropriate 
to evaluation questions, purposes, objectives and design 

* Select analyses appropriate to evaluation questions, purposes, 
objectives, data and design. 
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MONITORING 

o Identify specific activities to be monitored. 

o Select or develop monitoring forms. 

o Develop a schedule for activity monitoring. 

o Notify participants/program director of schedule. 

o Identify and train individual(s) responsible for selected monitoring 
activities. 

REPORTING 



0 


Identify recipients of final report. 


0 


Determine reporting mode (e.g., written, oral, combination). 


0 


Develop r.tH)rt format. 


0 


Obtain approval on mode and format. 


0 


Write draft report. 


♦ 


Have report reviewed by a sample of report recipients. 




Revise report. 


0 


Disseminate. 
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Program Evaluation Guide , Princeton: Educational Testing Services, 
1979. 

Sciiven, M. The Evaluation of Educational Goals, Instructional Procedures 
and Outcomes, 1972 (ERIC No, ED 079 394), 

Tyier, R, W, Basic Principles of Curriculum and Instruction, Chicago: The 
University of Chicago Press, 1971. 

Workbook on Program Evaluation , Princeton: Educational Testing Services, 
1979, 
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These materials were developed based upon materials by Educational Testing 
Service (ETSO They may be used in conjunction with the Evaluation Improvement 
Program, Workbook on Program Evaluation . 



Copyright (c) 1985 Educational Testing Service. All rights reserved. 
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State of Florida 
Department of Education 
Tallahaaiee, Florida 
Ralph D. Turiington, Commissioner 
AfSrmative action/equal opportunity employer 



FLORIDA: A STATE OF EDUCATIONAL DISTINCTION. "On a statewide average, 
educational achievement in the State of Florida will eaual that of the upper quartile of 
states within five years, as indicated by commonly accepted criteria of attainment." 
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